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(57) ABSTRACT 

Reception of digital multimedia data signals from multiple 
sources in a stream over a fixed bandwidth communications 
path may be accomplished by subscribing to a base layer of 
a first source and a base layer of a second source, and 
subscribing to an enhancement layer of the first source. Data 
signals corresponding to the subscribed layers of the first 
and second sources may be received in a stream over the 
fixed bandwidth communications path, output signals may 
be produced which correspond to the received data signals 
for the first source, and output signals may be produced 
corresponding to the received data signals for the second 
source. The output signals for the first source may be 
displayed in a first portion or window of a display, and 
output signals for the second source may be simultaneously 
displayed in a second portion or window of the display, 
thereby providing a picture-in-picture (PIP) display for 
streaming digital video. 

19 Claims, 5 Drawing Sheets 




104 



END ) 



SUBSCRIBE TO BASE 
LAYERS OF SOURCES 



SELECT PRIMARY SOURCE FOR 
ENHANCEMENT LAYERS 
. PROCESSING 



_L 



10 



SUBSCRIBE TO AUDIO STREAM 
CORRESPONDING TO SELECTED 
PRIMARY SOURCE 



PRODUCE OUTPUT SIGNALS FOR 
AUDIO STREAM AND ENHANCEMENT 
AND BASE LAYERS FOR PRIMARY 
SOURCE 



112 



PRODUCE OUTPUT SIGNALS 
FOR BASE LAYER OF AT LEAST 
ONE SECONDARY SOURCE 



114 




UNSUBSCRIBE ENHANCEMENT 
LAYERS OF PRIMARY SOURCE 



]20 



SUBSCRIBE TO 
ENHANCEMENT LAYERS 
OF SECONDARY SOURCE 



122 



SWITCH PRIMARY 
AND SECONDARY 
STREAM IDENTITIES 



02/02/2004, EAST Version: 1.4.1 



U.S. Patent 



Jan. 21, 2003 Sheet 1 of 5 



US 6,510,553 Bl 




02/02/2004, EAST Version: 1.4.1 



U.S. Patent Jan. 21, 2003 Sheet 2 of 5 



US 6,510,553 Bl 




02/02/2004, EAST Version: 1.4.1 



U.S. Patent 



Jan. 21, 2003 Sheet 3 of 5 



US 6,510,553 Bl 




io„ 




Kit 



CD 



CM 




Z 

LU ^ 

LU CC 
QUJ 

n 

z 

UJ 

m 



UJ cc 

QUJ 

& 

UJ 



LU ^ 
HI CC 

QUJ 

z 

111 



< 



GO 

in 



CD 

lu a: 



o 

<D 



< 
CD < 



at 
a) 



00 

in 



CD 

uj 

"5 



O 
CD 



< 

uj q: 

m 5 



CO 

in 



CD 

uj cc 

m «5 



o 

CD 



\(D 
CD 



"N.CVJ 
CD 



CO 

& 
u. 



a) 

z 
o 

t- 

< X 

9k 
2 £ 

15 0- 



LU 



O 

in 



-o 



2 
O 
O 



02/02/2004, EAST Version: 1.4.1 



U.S. Patent Jan.21,2003 Sheet 4 of S US 6,510,553 Bl 






SUBSCRIBE TO BASE 






LAYERS OF SOURCES 


108 






SELECT PRIMARY SOURCE FOR 


ENHANCEMENT LAYERS 




PROCESSING 





jL 



110 



SUBSCRIBE TO AUDIO STREAM 
CORRESPONDING TO SELECTED 
PRIMARY SOURCE 




PRODUCE OUTPUT SIGNALS FOR 
AUDIO STREAM AND ENHANCEMENT 
AND BASE LAYERS FOR PRIMARY 
SOURCE 



112 



X 



PRODUCE OUTPUT SIGNALS 
FOR BASE LAYER OF AT LEAST 
ONE SECONDARY SOURCE 



114 

V 



UNSUBSCRIBE ENHANCEMENT 
LAYERS OF PRIMARY SOURCE 



120 

j 



SUBSCRIBE TO 
ENHANCEMENT LAYERS 
OF SECONDARY SOURCE 



122 

jL . 



SWITCH PRIMARY 
AND SECONDARY 
STREAM IDENTITIES 



Figure 4 



02/02/2004, EAST Version: 1.4.1 



U.S. Patent Jan.21,2003 Sheet 5 of 5 US 6,510,553 Bl 



4 \ 



402 



PROCESSOR 



CACHE 



i408 



413 
J. 



GRAPHICS 
CONTROLLER 



404 

_z_ 



PROCESSOR BUS 



410 



BRIDGE/MEMORY 
CONTROLLER 



406 

_Z_ 



MEMORY 



INSTRUCTIONS 



DATA 



412 
J— 



FIRST I/O BUS 



414 



NETWORK 
CONTROLLER 



416 



DISPLAY 
DEVICE 
CONTROLLER 



418 



CAMERA 



428 





BUS 




BRIDGE 



420 
J— 



SECOND I/O BUS 



422 



DATA 
STORAGE 



424 



KEYBOARD 
INTERFACE 



425 
J- 



USER 
INPUT 
INTERFACE 



426 



AUDIO 
CONTROLLER 



FIG. 5 



02/02/2004, EAST Version: 1.4.1 



US 6,510,553 Bl 
1 2 

METHOD OF STREAMING VIDEO FROM with a purported maximum bandwidth of 56K bytes per 

MULTIPLE SOURCES OVER A NETWORK second (bps), the modem may serve as an impediment to 

providing a clearer or larger display of the video images. 
This limitation of a fixed bandwidth communications path 

BACKGROUND 5 ^ particularly acute when the client desires to retrieve 

1. Field streams from more than one signal source at a time. In some 

The present invention relates generally to processing of situations, a user may desire to display video data signals 

multimedia data signals and, more specifically, to manage- from at leasl «w° si S nal evicts on a single display, with the 

ment of streaming digital audio and video data signals. data &° m cach source being represented on a portion of the 

2 Descriotion 10 ^isp^y- ^ e display of two independent video streams, each 

J. , . . , in a portion of a display, is widely used for analog television 

Digital multimedia data signals may be communicated ^ ^ ^ ^ .. picture . m . a . picture » (pi P) feature of many 

over a network from a signal source to a client system for ^ televisions HowcV6r> for Malog te i cvisionSi 6ach 

viewing and hstenmg by a user it the client site. In one video stn . am fc receive( , fc a ^ ^ oQ R 

scenano, a user requests the retocval of a sequence of audio 15 communications th and therefore the bandwidth of a 

and video data signals called a stream from the signal - , ■ V t , ? , , t c ^ 

... . B . L single communications path for the dual sources is not fixed, 

source, which may be a server system communicatmg with ° . . c , 

the client system over the network. When the audio and „ S.mulUneot^ reception of two or more streams over a 

video data signals making up the stream are received at the fixed ; ?T \ d f* l <*> mm ™ c ^ 0BS 

client, these multimedia data signals are processed to pro- 20 result m bandwidth contention be ween the streams. Without 

due* output data for the user in the form of sounds produced s ° me wa y 10 P nontl2e f en ' s rece P u ° n °, f 

. nna „ r a i , , 0 ^ 0 ,i . • o „ oo j- nU ; fil j „ a the streams may contend for the available, limited band- 

by one or more loudspeakers and images displayed on a 3 ... ' . 



display. 



width. The data packets from the streams are thus received 

n „ . , • ina non-deterministic manner. The result may be that the 

The concept of streaming multimedia mcludes a mulli- ^ ,. , ^ t . ^ a~ j a j ,„ tUa 

r =*. , . reception and processing 01 the streams is degraded to the 

media source sending a multimedia stream to a multimedia M in , whcre Qonc rf ^ streams ^ significant 

recipient for real-time presentation to a user. That is, stream- ^ informalion t0 ^ user TypicaUyj the user cannot 

mg can be viewed as a chsplay/play model as opposed to ^ which strcam to rcceivc riorit in ^ communica . 

a more traditional download and display/play model, tions fa tQ ensure ^ >( ^ Qne of ^ strelms CQ 

where the enure stream is collected by the recipient prior to e fa da , a d b for a ^ tQ (h6 user , n 

presentation. Successful streaming is based on the premise 30 additi rece tion of audio data signlls for a stream may 5e 

that the multimedia preparation (e.g., compression) and the d ded such ^ (he au(Jio ou becomcs ^ 

transmission mechan^m can feed the rectptent at a rate fast J*^ and undesirable for the user 

enough for the multimedia stream to be presented in real- , . j . • . iL iLiJJ 

% t • 1 u a: ■ „!] „„t n f t u a What is needed is an approach or technique that addresses 

time. Streaming may involve buffering a small amount or the f . m 

stream prior to presentation, but the amount of buffering is 35 mc tore S oin g problems. 

typically small compared to the length of the entire stream. SUMMARY 

The rationale for buffering is typically to smooth out jitter 

inherent in network-based transmission mechanisms (e.g., An embodiment of the present invention is a method of 

packet-based transports, etc.)- receiving digital multimedia data signals from multiple 

r. . , u * • „ u- a- » ~a .0 sources over a fixed bandwidth communications path using 

One difference between streaming multimedia and mul- 40 , . , . r ^ • i - t i . -i * 

1 * ,1 , 1 1 , 1 j j f ♦ ■ •* *• * \ multiple layers of a stream. The method includes subscribing 

ti media that has been downloaded (in part or in its entirety) £ , J _ „ a . . ° 

tU t t „■ *w ma A' n to a first layer of a first source, a first layer of a second 

prior to presentation is that streaming multimedia quality * t t * t c ^ ... 

i C , j , , e , \ • j * -At source, and a second layer of the first source. The method 

(for example, video clarity and trame-rate) is determined, at ■ • j . ■ i j- ..u 

1 . • - u *u u a ,-a*u «f fu~ m ^u» further includes receiving data signals corresponding to the 

least in part, by the bandwidth of the transmission mecha- L n_ j » r.L ^. j j -.u 

f-» t 1 jj i*- j * .1 v* f *l „ ^ subscribed layers of the first and second sources in the 

msm. For downloaded multimedia, the quality of the pre- 45 j 

sentation is typically unaltered from what it had been slr£ \ am over * e fixed ^ Wldth communications path and 

authored as; however, the download times are affected by the Producing output signals corresponding to the received data 

bandwidth of the transmission channel. Asecond difference S1 g nals for the &sl source and the second source - 

between the two models is that the download model is not Another embodiment of the present invention is a method 

workable for "live" content. In a display/play model of a 50 of receiving multiple, independent digital multimedia 

digital multimedia streaming system, the size of the dis- streams over a bandwidth communications path. The 

played images and the overall quality of the audio and video method includes subscribing to first and second streams, the 

presented to the user are limited at least in part by the stream corresponding to a first multimedia content and 

bandwidth of the communications path between the signal me second stream corresponding to a second multimedia 

source and the client. In some systems, a sufficient amount 55 content. The method further includes concurrently receiving 

of audio and video data signals is received to display dala signals corresponding to the streams over the fixed 

full-motion video and concurrent audio to the user in a bandwidth communications path, and producing output sig- 

real-time (or streaming) mode, although the size and quality nals corresponding to the received data signals for the first 

of the displayed image may be inadequate or undesirable for slrcam and for me s*™ 6 stream, 

some uses. If the bandwidth of the communications path 60 BRIEF DESCRIPTION OF THE DRAWINGS 
could be expanded, then a larger image, a clearer image, or 

a smoother set of images could be presented to the user. The features and advantages of the present invention will 

However, in most systems, the communications path become apparent from the following detailed description of 

between the signal source and the client is a fixed, relatively the present invention in which: 

low bandwidth communication channel having at least one 65 FIG. 1 is a diagram of a system for multiple source 

bottleneck. For example, when the network is the Internet real-time video streaming over a network according to one 

and the client is a computer system employing a modem embodiment of the present invention; 
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FIG. 2 is a diagram of a system for dual source real-lime is not limited in this respect. The connection between 

video streaming over a network according to another communications network 14 and multimedia access and 

embodiment of the present invention; display device 12 represented by communications path 16 

FIG. 3 is a diagram illustrating bandwidth partitioning of ma V ^ any link with a predefined maximum data 

a communications path and corresponding video images s bandwidth, such as for example modem-based dial-up 

m » • ... r connections, various types of digital subscriber line (DSL) 

according to one embodiment of the present invention; • ui j *- • i 

6 r connections, cable modem connections, wireless 

FIG. 4 is a flow diagram of controlling bandwidth parti- connections, or network interface card (NIC) based intranet 

tioning of multiple layers of a stream according to an connections, although the invention is not limited in this 

embodiment of the present invention; and ^ respect For purposes of this embodiment, only a fixed 

FIG. 5 is a diagram illustrating a sample system suitable maximum bandwidth for communications path 16 is 

to be programmed according to an embodiment of a method stipulated, however, the actual data rate for the communi- 

for streaming video from multiple sources over a network. cations path may vary with time without exceeding the 

nccPDlimAW maximum value. Additionally, the communications path 

DETAILED DESCRIPTION ^ supports bi-directional data flow (although not necessarily 

An embodiment of the present invention includes a symmetrical in terms of bandwidth for each direction of data 

method whereby a client receives at least two real-time flow). 

independent streams or at least two layers of a stream A plurality of signal sources Source 1 18, Source 
containing digital video data signals over a fixed bandwidth 2 20, ... , Source N 22 may be coupled to communications 
communications path with user control of which stream or 2 o network 14 to serve as multimedia content repositories 
layer is "in focus" as the primary source of display data and accessed by multimedia access and display device 12. In one 
which stream or layer is a secondary source for display data. embodiment, the number of signal sources actively serving 
In one embodiment, the method may be accomplished using multimedia access and display device 12 may be two. In 
scaleable audio and video coding. An embodiment of the other embodiments, any number of signal sources may be 
present invention provides a picture- in -picture (PIP) capa- 2 s active and available for serving multimedia data streams to 
bility to live video streaming over a communications chan- multimedia access and display devices at any time. Each 
nel with limited bandwidth (such as a dial-up Internet signal source stores or has access to at least a portion of a 
connection, for example). This enables new functions for digital multimedia content represented as a sequence of 
network streaming video applications (such as Internet audio and video data signals called streams. The multimedia 
streaming, for example) and may be used to improve stream- 30 content may be stored at a signal source site in a main 
ing applications that incorporate a greater degree of user- memory, a secondary memory, or a storage medium such as, 
interactivity and improved usability. for example, a CD-ROM, a digital versatile disk (DVD), or 
In one embodiment of the present invention, a client other suitable apparatus. As used herein, a stream comprises 
system may receive a single stream of video data signals, the a plurality of audio and/or video data signals and the stream 
single stream comprising multiple layers, each layer being 35 mav comprise multiple layers. Each layer of a stream may 
communicated from a different video data source and con- comprise data from a source. The multimedia content may 
taining video data signals of a portion of a selected multi- also be acquired in real-time from live audio and/or video 
media content. The client system may present video data sources such as, for example, a video camera and a micro- 
signals from a first source for display in a primary portion of phone. 

a display while simultaneously displaying video data signals 40 When a user of multimedia access and display device 12 
from a second source in a secondary portion of the display. desires to hear and/or view selected multimedia content, the 
At the direction of the user via a graphical user interface, the user indicates his or her selection to multimedia access and 
client system may then switch the bandwidth partitioning of display device 12 in any one of many well-known ways, 
the stream so that video data signals from the first source such as, for example, by using a pointing device or keyboard 
may be displayed in the secondary portion of the display and 45 and a graphical user interface and/or a browser application, 
video data signals from the second source may be displayed In one embodiment, specific desired multimedia content 
in the primary portion of the display. The video data signals may be identified by a uniform resource locator (URL) 
from each of the sources may be carried by layers of the and/or identifiers according to the well-known hyper text 
stream and may be received over the same limited band- transport protocol (HTTP), although the invention is not 
width communications path. However, the allocation of the so limited in scope in this respect. Multimedia access and 
bandwidth between the primary and secondary sources display device 12 interacts with a signal source having the 
changes according to the user selection. desired content to initiate the sending of a stream of audio 
FIG. 1 is a diagram of a system 10 for multiple source and/or video data signals representing the desired content 
real-time video streaming over a network according to one over communications network 14. Upon reception, multi- 
embodiment of the present invention. A multimedia access 55 media access and display device produces audio and/or 
and display device 12 may be coupled to a communications video output signals to the user based at least in part on the 
network 14 by a communications path 16. Multimedia received stream. In the case of video output data, the video 
access and display device 12 may be any device for may be displayed on at least a first portion of a display. In 
receiving, processing and displaying audio and video the case of audio output data, the audio may be produced by 
streams, such as, for example, a personal computer (PC) or 60 at least one loudspeaker (not shown), 
other computer system having a display, a set-top box, an Subsequent to initiation of transmission of a stream of 
Internet appliance, a JAVA terminal, an information kiosk, or video data signals from a first source, the user may desire to 
a digital television (DTV), although the invention is not substantially simultaneously receive data signals from one 
limited in this respect. Communications network 14 may be or more additional sources. Additional sources may be 
any network for coupling computing devices such as, for 65 initiated and transmitted in a manner similar to the first 
example, a local area network (LAN), a wide area network source. However, additional processing may be employed to 
(WAN), an intranet, or the Internet, although the invention ensure that multiple sources are authored by multimedia 
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content providers, transmitted by signal sources, and Annex O of the H.263+ specification. Annex O defines three 
received by multimedia access and display devices in such scalability features termed temporal, signal to noise ratio 
a way as to provide substantially simultaneous display of the (SNR), and spatial scalability. With temporal scalability, an 
data signals from multiple sources on different portions of a encoder generates bi-directionally predicted frames (also 
display at a quality that is acceptable to the user. 5 called true B frames). These frames support scalability in the 
Such processing may be best understood with reference to sense that they may be discarded without hindering the 
a specific example system having two active signal sources. decoding of surrounding frames in the stream. For example, 
However, the invention is not limited in scope to having only these frames maybe dropped at the decoder side to reduce 
two active signal sources and in some embodiments any the display frame rate (thus scaling processor usage), or they 
number of active signal sources may be handled according 1(J may be dropped at the encoder side of a multi-point 
to the teachings of the present invention. FIG. 2 is a diagram connection, to free up bandwidth on low capacity commu- 
of a system 30 for dual source real-time video streaming nications channels. True B frames also offer the advantage 
over a network according to another embodiment of the that they may be compressed more efficiently than predicted 
present invention. In this embodiment of the present (p) frames, at approximately the same quality level, 
invention, there are two active signal sources, Source A 32 SNR scalability is a technique where the error introduced 
and Source B 34. Generally, these sources may be any b compression ^ iteclf encoded and transmitted as a 
systems having access to multimedia content. In one m63+ encoded icture M ^ receivi cnd> ^ 
embodiment, signal sources A 32 and B 34 may comprise en £ ancement { picture F may be decoded and added back 
server computer systems coupled to computer network 36. fc b ^ * ^ ^ n . fa tf . 
In another embodiment, the signal sources comprise a single , t F , 7 , . _? 4 y lu ^ a & v l " " JU 
server system. In one embodiment, the computer network 20 * base Pfctiue was decoded. Since compression of the error 
may be the Internet. In various embodiments, the signal P icture IS lts f loss ^ the remaining error after that corn- 
sources may be structured to provide streaming layered P resslon ma y be interpreted as yet another picture, and may 
video streams authored to conform to International Tele- De compressed and transmitted as yet another enhancement 
communications Union (ITU) Video Coding for Low Bit layer. The H.263+ specification supports up to 14 enhancc- 
Rate Communication, Recommendation H.263, version 2, 25 ment lavers ^ manner > wit h the bandwidth allocated to 
1998 (also known as "H.263+"), and to accommodate net- each layer by an authoring application, 
work connections up to 56 Kbps. The H.263+ specification Spatial scalability is a variation of SNR scalability, where 
provides for scaleable video coding by allowing the total an enhancement layer may be four times as large (twice the 
bandwidth of a single stream to be divided into separate dimensions) as the lower layer. This allows a common 
layers of data for a given video signal source, each layer 30 interchange format (CIF) Input sequence, for example, to be 
corresponding to a different quality level associated with the encoded as both a base layer quarter common interchange 
video. Further information relating to layered streams may format (QCIF) sequence and an enhancement layer CIF 
be found in Annex O of the H.263+ specification. The sequence. Frame rate scalability is another feature available 
layered video encoding strategy employed by H.263+ imple- with SNR and spatial scalability. With frame rate scalability, 
mentations provides the capability to subscribe to primary 35 an enhancement layer may comprise more frames per sec- 
and secondary layers of a stream at different data rates, ond than a base layer. 

thereby enabling a PIP mode of operation. i n a n embodiment of the present invention, the video 
Of course, the use of the H.263 specification to define the stream may comprise a plurality of layers including a base 
format of the stream and the 56 Kbps limit on the network layer (LO) at a data rate of 8 K BPS, supporting 5 frames per 
connection is only representative of various embodiments of 40 second (fps) of a 80 pixel by 60 pixel video image and an 
the present invention and the invention is not limited in intracoded frame-predicted frame-predicted frame (IPP) 
scope in these respects. profile; an enhancement layer 1 (LI, reference layer LO) at 
A client system 38 (such as a personal computer system, a data rate of 8 K BPS, providing a SNR layer, supporting 
for example) may be coupled to network 36 over a com- 5 fps of an 80 pixel by 60 pixel video image and an IPP 
munications path 40 as a multimedia access and display 45 profile; an enhancement layer 2 (L2, reference layer LI) at 
device. Communications path 40 comprises a fixed band- a data rate of 10 K BPS, providing a spatial layer, supporting 
width communications link (for example, a 56 Kbps link). 5 fps of a 160 pixel by 120 pixel video image and an IPP 
Client system 38 may comprise a decoder 42 for decoding profile; and an enhancement layer 3 (L3, reference layer L2) 
H.263+ layered streams received by the client and a graphi- at a data rate of 10 K BPS, providing a temporal layer, 
cal user interface 44 for displaying the video data signals of 50 supporting 15 fps of a 160 pixel by 120 pixel video image 
the streams to the user and accepting user input selections. and a multiple temporal scalability frame (BBBB) profile. 
Graphical user interface 44 may be used to select a desired The scalability configuration described herein is an example 
multimedia content as a primary source for a first layer of a of one embodiment and not the only stream configuration 
stream and another multimedia content as a secondary that may be used with the present invention. Furthermore, 
source for a second layer of the stream. Audio data associ- 55 the data rates, frame rates, and image sizes discussed above 
ated with both the primary and secondary sources may be are but one example of a layered stream configuration. One 
compressed for a resulting constant data rate of 8 Kbps in skilled in the art will appreciate that these parameters may 
this embodiment. The audio data signals may be transmitted be changed in other embodiments and this specific example 
to the client system on an audio stream. The configurations will not serve to limit the scope of the present invention, 
of layers used to encode video streams may be stored in 60 According to this configuration of layers, the total band- 
catalog 46 that may be transmitted to client system 38 at the width used by the client system subscribing to the base layer 
time a multimedia transfer is initiated. The catalog may from one source and all base and enhancement layers from 
comprise subscription options and information regarding another source fits into the 56 K BPS communications path 
whether a particular stream may be used in a PIP mode of provided by communications path 40. This assumes that 
operation. 65 only one audio stream is subscribed to at any given point in 
In one embodiment, the individual video streams may be lime. Generally, multiple audio streams are mutually exclu- 
authored to support configurations in accordance with sive and the audio data signals communicated to the client 
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system correspond to the primary source. In one exist, then multiple source streaming processing ends at 

embodiment, the audio stream may be included as a separate block 104. If at least one base layer exists for each source, 

layer within the stream. the client system subscribes to the base layers of the sources 

In this embodiment, the low bandwidth base layer (LO) at block 106, with each source providing data for a base 

may be used as a "preview" layer to provide PIP capability. 5 layer of the video stream. Next, at block 108, a primary 

FIG. 3 is a diagram illustrating the bandwidth partitioning of source may be selected for subscription and subsequent 

a communications path and corresponding video images processing of the enhancement layers associated with the 

according to one embodiment of the present invention. At primary source. At block 110, the audio data stream for the 

time Tl 50, a user of client system 38 selects the multimedia selected primary source may be subscribed to for subsequent 

content from signal source A 32 as the primary source of the 10 reception and production of audible sounds. After these 

stream and the multimedia content from signal source B 34 actions, the video stream being received by the client system 

as a secondary source of the stream. As a result of this comprises base layers for at least two signal sources and 

selection, video data from the primary source may be shown enhancement layers for one signal source. The client system 

on a display in a first portion 52 of the screen, and video data may also receive an audio stream corresponding to the signal 

from the secondary source may be shown on the display in 15 source of the enhancement layers. At block 112, the output 

a second portion 54 of the screen, thereby providing PIP signals for the audio stream and enhancement layers and 

capability. The first portion may be larger than the second base layer of the primary source may be produced for 

portion because more information for display is being reception by the user of the client system. This action may 

received for the primary source as compared to the second- comprise rendering the video data signals from the base and 

ary source. In this example, the secondary source data ("B") 20 enhancement layers of the video stream on a display and 

is shown in a rectangle in the lower right comer of the generating audible sounds through at least one loudspeaker 

screen. However, this is only an example and other sizes, from the audio data signals of the audio stream. The audible 

shapes, and placements may be used. sounds may be generated in synchronization with the dis- 

In the configuration of layers at time Tl, the client system played images from the base and enhancement layers. The 

subscribes to the enhancement layer 56 and a base layer 58 25 video data signals may be rendered to a first portion of the 

of signal source A 32, and a base layer 60 of signal source display. At block 114, the output signals for the base layer of 

B 34. Both base layers and the enhancement layers are secondary source may be produced for reception by the user 

transmitted along fixed bandwidth communications path 40. of the client system. This action may comprise rendering the 

The two decoded video sequences from these layers may be video data signals from the base layer to a second portion of 

displayed in the PIP format as shown. The client system also 30 the display. 

subscribes to an audio stream corresponding to the primary When the client system is in this configuration, the user 

source (at time Tl, this is from signal source A). At some may view a continual stream of data from a first multimedia 

time T2 62, the user may decide to switch the focus between content in a first portion of the display and a second 

the two windows. This may be indicated by double clicking multimedia content in a second portion of the display (e.g., 

a computer mouse button when a mouse pointer is over the 35 a PIP display). At some point in time, the user may desire to 

PIP area, for example, although any method of indicating change the selection of the primary source. For example, this 

input to the graphical user interface may be employed. As a could be accomplished by double clicking a mouse button 

result of the switch indication, the client changes the current with a mouse pointer over an area of the display or an 

subscription of sources to accept only the base layer 58 of equivalent keyboard action. When such an action is taken, 

signal source A, but to accept the base layer 60 and the 40 the subscription for the selected portion of the display may 

enhancement layers 64 of signal source B. This results in a be changed to become the new primary source. At block 116, 

change to the display whereby video data signals from signal it may be determined if changing of the sources is desired 

source B are the primary source shown in the first portion 52 (e«g- ; by recognizing an appropriate input event). If no 

of the screen and video data signals from signal source A are change is indicated, then processing continues by producing 

the secondary source shown in the second portion 54 of the 45 more output signals from the received audio and video 

screen. The switch may be delayed by a small time interval streams at block 112. If a change is indicated, then the 

so that subscription changes may be synchronized with key enhancement layers of the primary source are unsubscribed 

frames in the stream that are assumed to be present at regular at block 118. At block 120, the client system subscribes to 

intervals. At the time of the switch, the audio subscription the enhancement layers of the secondary source. At block 

may also be changed to maintain the association with the 50 122, the identities of the primary and secondary sources are 

appropriate source selected as the primary source. switched or otherwise exchanged (i.e., the primary source 

At time T3 66, a user may desire to switch back to signal becomes the secondary source and vice versa). Processing 

source A as the primary source. As a result of the switch continues with block 110. 

indication, the client changes the current subscription of In an alternate embodiment of the present invention, the 

sources to accept only the base layer 60 of signal source B, 55 digital multimedia content may be authored as multiple 

but to accept the base layer 58 and the enhancement layers independent streams as opposed to a single stream having 

56 of signal source A. This results in a change to the display multiple layers. In one embodiment, the multiple indepen- 

whereby video data signals from signal source A are again dent streams may be authored according to the Smart- 

the primary source shown in the first portion 52 of the screen Stream™ streaming multimedia content technology com- 

and video data signals from signal source B are again the 60 mercially available from Real Networks, Inc., although the 

secondary source shown in the second portion 54 of the present invention is not limited in scope in this respect, 

screen. Unlike the layered approach used by the H.263+ specifica- 

FIG. 4 is a flow diagram of controlling bandwidth parti- lion to address bandwidth (and processor) scalability, Smarl- 

tioning of multiple layers of a stream according to an Stream™ technology is a non-additive method of encoding 

embodiment of the present invention. After start 100, a client 65 a video sequence into a set of independent streams by 

system determines if base layers exist for at least two signal choosing a different set of encoding parameters for each 

sources at block 102. If a single base layer or no base layers stream. The streams are independent in the sense that each 
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stream may be decoded without any information from the systems comprising at least one processor, a data storage 

other streams. From the multimedia content author's point system (including volatile and non-volatile memory and/or 

of view, the author may choose to lake the same video storage elements), at least one input device, and at least one 

content and encode it with different encoding parameters output device. Program code may be applied to input data to 

(e.g., data rate, frame rate, video size), storing all the 5 perform the functions described herein and generate output 

resultant streams in the same file. For example, one stream information. The output information may be applied to one 

may comprise base layer video data signals while another or more output devices, in known fashion. For purposes of 

stream may comprise enhancement layer video data signals. this application, a processing system includes any system 

The client system may switch between the different avail- that has a processor, such as, for example, a digital signal 

able streams under conditions of changing bandwidth (for 10 processor (DSP), a microcontroller, an application specific 

example, the client system may switch subscription from a integrated circuit (ASIC), or a microprocessor, 

high data rate stream to a lower data rate stream if the Tn e programs may be implemented in a high level pro- 

transmission channel bandwidth decreases suddenly) and cedural or object oriented programming language to com- 

processor load (for example, the client system may switch municate with a processing system. The programs may also 

from a high frame rate stream to a lower frame rate stream is be implemented in assembly or machine language, if 

to reduce the amount of processor usage in decoding the desired. In fact, the invention is not limited in scope to any 

video data signals). particular programming language. In any case, the language 

Referring back to FIG. 2, two active sources 32, 34 may may be a compiled or interpreted language, 

be used, with each source streaming an independent H.263+ The programs may be stored on a storage media or device 

stream according to SmartStream™ video stream technol- 20 (e.g., hard disk drive, floppy disk drive, read only memory 

ogy. These streams may be authored to accommodate con- (ROM), CD-ROM device, flash memory device, digital 

nections up to a fixed data rate, such as 56 K BPS, for versatile disk (DVD), or other storage device) readable by a 

example. The client system 38 may be a personal computer general or special purpose programmable processing 

using software that can decode H.263+ SmartStream™ system, for configuring and operating the processing system 

video streams (e.g., decoder 42) and has a user interface 25 when the storage mcc ji a or device is read by the processing 

(e.g., graphical user interface 44) to allow the user to select system to perform the procedures described herein. Embodi- 

one of the streams as the primary (main) stream and the ments of the invention may also be considered to be imple- 

other as a secondary (preview) stream. Audio associated mented as a machine-readable storage medium, configured 

with both streams in one example may be compressed at 8 f or use with a processing system, where the storage medium 

K BPS (constant bit rate streams). 30 S o configured causes the processing system to operate in a 

The individual video streams representing digital multi- specific and predefined manner to perform the functions 

media content may be authored to support a configuration described herein. 

such as one having a first stream operating at 8 K BPS, and An example of one such type of processing system is 

supporting a 5 fps data rate of a 80 pixel by 60 pixel video ^ shown in FIG. 5. Sample system 400 may be used, for 

(e.g., a preview or base stream), and a second stream example, to execute the processing for methods in accor- 

operating at 28 K BPS supporting a 15 fps data rate of a 160 dance with the present invention, such as the embodiment 

pixel by 120 pixel video (e.g., a main or enhancement described herein. Sample system 400 is representative of 

stream). The SmartStream™ configuration described above processing systems based on the PENTIUM®, PENTIUM® 

is an example and not the only one that may be used for ^ p ro> an d PENTIUM® II microprocessors available from 

purposes of the present invention. In this embodiment, a low 4 Intel Corporation, although other systems (including per- 

bandwidth stream may be used as the preview stream and the sona i computers (PCs) having other microprocessors, engi- 

total bandwidth used by a client system subscribing to the neering workstations, set-top boxes and the like) may also be 

preview stream from one source and the main stream from used. In one embodiment, sample system 400 may be 

the second source still fits in a fixed bandwidth communi- ^ executing a version of the WINDOWS™ operating system 

cations channel (assuming only one audio stream is sub- available from Microsoft Corporation, although other oper- 

scribed to any given point in time). a ii ag systems and graphical user interfaces, for example, 

The configuration used to encode each video stream may may also be used, 

be stored in a catalog 46 that is transmitted to an application fig. 5 is a block diagram of a system 400 of one 

on the client system 38 at the time the multimedia transfer 50 embodiment of the present invention. The computer system 

is initiated. This allows the client system to detect at the start 400 includes a processor 402 that processes data signals. The 

of the transfer what the subscription options are and to processor 402 may be a complex instruction set computer 

determine if a particular stream can be used in the manner (CISC) microprocessor, a reduced instruction set computing 

described by an embodiment of the present invention. (RISC) microprocessor, a very long instruction word 

In the preceding description, various aspects of the 5S (VLIW) microprocessor, a processor implementing a corn- 
present invention have been described. For purposes of bination of instruction sets, or other processor device, such 
explanation, specific numbers, systems and configurations as a digital signal processor, for example. FIG. 5 shows an 
were set forth in order to provide a thorough understanding example of an embodiment of the present invention imple- 
of the present invention. However, it is apparent to one mented as a single processor system 400. However, it is 
skilled in the art that the present invention may be practiced 60 understood that embodiments of the present invention may 
without the specific details. In other instances, well-known alternatively be implemented as systems having multiple 
features were omitted or simplified in order not to obscure processors. Processor 402 may be coupled to a processor bus 
the present invention. 404 that transmits data signals between processor 402 and 

Embodiments of the present invention may be imple- other components in the system 400. 

mented in hardware or software, or a combination of both. 65 System 400 includes a memory 406. Memory 406 may be 

However, embodiments of the invention may be imple- a dynamic random access memory (DRAM) device, a static 

mented as computer programs executing on programmable random access memory (SRAM) device, or other memory 
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device. Memory 406 may store instructions and/or data may reside in another device such as a bus controller or other 

represented by data signals that may be executed by pro- controller device. Keyboard interface 424 allows coupling 

cessor 402. The instructions and/or data may comprise code of a keyboard to system 400 and transmits data signals from 

for performing any and/or all of the techniques of the present a keyboard to system 400. A user input interface 425 may be 

invention. Memory 406 may also contain additional soft- 5 coupled to the second I/O bus 420. The user input interface 

ware and/or data (not shown). A cache memory 408 may may be coupled to a user input device, such as a mouse, 

reside inside processor 402 that stores data signals stored in joystick, or trackball, for example, to provide input data to 

memory 406. Cache memory 408 in this embodiment speeds the computer system. Audio controller 426 may be coupled 

up memory accesses by the processor by taking advantage of to the second I/O bus 420. Audio controller 426 operates to 

its locality of access. Alternatively, in another embodiment, J(J coordinate the recording and playback of audio signals. A 

the cache memory may reside external to the processor. bus bridge 428 couples first I/O bridge 412 to second I/O 

A bridge/memory controller 410 may be coupled to the brid S e 420 ^ bus brid S e operates to buffer and bridge data 

processor bus 404 and memory 406. The bridge/memory si S mls between the first I/O bus 412 and the second I/O bus 



controller 410 directs data signals between processor 402, 



420. 



memory 406, and other components in the system 400 and „ Embodiments of the present invention are related to the 
bridges the data signals between processor bus 404, memory of ***** 400 t0 receive multiple digital multimedia 
406, and a first input/output (I/O) bus 412. In some si ™™ j; According to one embodiment reception of user- 
embodiments, the bridge/memory controller provides a f lectabl K e rnuluple digital mulUmedia streams may be per- 
, . ' i- , , . £ ji* t formed by the system 400 in response to processor 402 
graphics port for coupling to a graphics controller 413. Id executing sequences of instruction in memory 406. Such 
this embodiment, graphics controller 413 interfaces to a M instructions may be read into memory 406 from another 
display device (not shown) for delaying images rendered compu ter-readable medium, such as data storage device 422, 
or otherwise processed by the graphics controller 413 to a Qr from another gaum ^ (he ne(work controller 414 for 
user. The display device may comprise a television se^ a exanl le Execution of the sequences of instructions causes 
computer monitor, a flat panel display, or other suitable processor 402 to receive multiple digital multimedia streams 
display device. 25 according to embodiments of the present invention. In an 
First I/O bus 412 may comprise a single bus or a com- alternative embodiment, hardware circuitry may be used in 
bination of multiple buses. First I/O bus 412 provides place of or m com bination with software instructions to 
communication links between components in system 400. A implement embodiments of the present invention. Thus, the 
network controller 414 may be coupled to the first I/O bus preS ent invention is not limited to any specific combination 
412. The network controller links system 400 to a network 30 of hardware circuitry and software, 
that may include a plurality of processing systems (not ^ elements 0 f system 400 perform their conventional 
shown in FIG. 3) and supports communication among mnctions we i]-known in the art. In particular, data storage 
various systems The network of processing systems may devjce m b(J ^ tQ ide long . tenn storage for , he 
comprise a local area network (LAN), a wide area network executable inslruct ion« and data structures for embodiments 
(WAN), the Internet, or other network. In some 35 of mctho ds of receiving streaming multimedia data signals 
embodiments, a display device controller 416 may be from mulliple sources in accord ance with the present 
coupled to the first I/O bus 412. The display device con- mvemioD , WD6reas memory 40 6 is used to store on a shorter 
trailer 416 allows coupling of a display device to system 400 lem basjs me executa51e i nstruc tions of embodiments of the 
and acts as an interface between a display device (not methods for receiving streaming multimedia data signals 
shown) and the system. The display device may comprise a 40 from muhi ]e in accordance ^ me present inven- 
telev>sion set, a computer monitor, a flat panel display, or ^ ^ execution by processor 402. 
other suitable display device. The display device receives ^ ^ ^ ^ refcrence to 
data signals trom processor 4UZ through display device mustrative embodiments , , his 

description is not intended to 

controller 416 and displays information contained in the data , ,. ... . ,. c ,. f ,. 

, be construed in a limiting sense. Various modifications oi the 

signals to a user of system 400. 45 ... ... ... , „ . ,■ , e 

° ' , illustrative embodiments, as well as other embodiments of 

In some embodiments, camera 418 may be coupled to the , he invention( which are appar ent to persons skilled in the art 

first I/O bus. Camera 418 may comprise a digital video (o which the ^eniovs ^ are deemed to lie the 

camera having internal digital video capture hardware that irfl an(J of , he iovention . 

translates a captured image into digital graphical data. The What is claimed is- 

camera may comprise an analog video camera having digital 50 L A method of recei ving digital multimedia data signals 

video capture hardware external to the video camera for froin muhi k sources m & s , ream over a 

digitizing a captured image. Alternatively, camera 418 may fixed bandwidlh ^^^1^ palh comprising: 

comprise a digital still «mera or an analog still camera subscribing t0 a first layer of a first a flrst , ayer of 

coupled to image capture hardware. A second I/O bus 420 ° J , , f ~ 4 

r • • i . . ■ c i t - t a second source, and a second layer of the first source; 

may comprise a single bus or a combination of multiple 55 . . , A . ' J . ^ . . 

buses. The second I/O bus 420 provides communication re f lvm S dala signals corresponding to the subscribed 

links between components in system 400. A data storage ^yct* ° f * c ^ and sccond sourccs m ^ stream ovcr 

device 422 may be coupled to the second I/O bus 420. The the bandwidth communications path; 

data storage device 422 may comprise a hard disk drive, a producing output signals corresponding to the received 

floppy disk drive, a CD-ROM device, a flash memory 6 o data si S nals for & c first source and for lhc 

device, or other mass storage device. Data storage device source; 

422 may comprise one or a plurality of the described data displaying the output signals for the first source in a first 

storage devices. portion of a display; 

A keyboard interface 424 may be coupled to the second * substantially simultaneously displaying the output signals 

I/O bus 420. Keyboard interface 424 may comprise a 65 for the second source in a second portion of the display; 

keyboard controller or other keyboard interface device. unsubscribing the second layer of the first source; and 

Keyboard interface 424 may comprise a dedicated device or subscribing to a second layer of the second source. 
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2. The method of claim 1, wherein the first layer of the 
first source comprises a base layer of the stream, the second 
layer of the first source comprises an enhancement layer of 
the stream, and the first layer of the second source comprises 
another base layer of the stream. 5 

3. The method of claim 1, further comprising: 
subscribing to a third layer of the first source, the third 

layer comprising audio data signals; 
receiving data signals corresponding to the third layer of a0 

the first source in a second stream over the fixed 

bandwidth communications path; and 
producing audio output signals corresponding to the 

received data signals for the third layer. 

4. The method of claim 1, wherein the second layer of the 35 
second source comprises an enhancement layer of the sec- 
ond stream. 

5. The method of claim 1, further comprising: 
displaying the output signals for the first source in the 

second portion of a display; and 
substantially simultaneously displaying the output signals 
for the second source in the first portion of the display. 

6. The method of claim 5, further comprising: 
unsubscribing the second layer of the second source; and 25 
subscribing to the second layer of the first source. 

7. The method of claim 6, further comprising: 
displaying the output signals for the first source in the first 

portion of a display; and 
substantially simultaneously displaying the output signals 
for the second source in the second portion of the 
display. 

8. An article comprising a machine readable medium 
having a plurality of machine readable instructions, wherein 
when the instructions are executed by a processor the 
instructions subscribe the processor to receive a first layer of 
a first source, a first layer of a second source, and a second 
layer of the first source, control the reception of data signals 
corresponding to the subscribed layers of the first and 
second sources in a stream over a fixed bandwidth commu- 
nications path, produce output signals corresponding to the 
received data signals for the first source and the second 
source, display the output signals for the first source in a first 
portion of a display, substantially simultaneously display the 



30 



35 



subscribing to base layers of the multiple sources; 
subscribing to an enhancement layer of a selected one of 
the sources; 

receiving the base layers and the enhancement layer in the 
layered stream over the fixed bandwidth communica- 
tions path at the client system; 

displaying the base layer and the enhancement layer of the 
selected source in a first portion of a display of the 
client system; 

substantially simultaneously displaying the base layer of 

at least one non-selected source in at least one other 

portion of the display; and 
unsubscribing the enhancement layer of the selected 

source and subscribing to an enhancement layer of a 

newly selected source. 

13. The method of claim 12, further comprising subscrib- 
ing to an audio layer of the selected source, receiving the 
audio layer at the client system, and producing audio output 
signals in synchronization with the displaying of the base 
and enhancement layers of the selected source. 

14. The method of claim 12, further comprising: 
receiving the base layers of the sources and the enhance- 
ment layer of the newly selected source; 

displaying the base layer and the enhancement layer of the 
newly selected source in the first portion of the display; 
and 

substantially simultaneously displaying the base layer of 
at least one non-selected source in at least one other 
portion of the display. 

15. The method of claim 12, further comprising control- 
ling subscription of the enhancement layer by a user of the 
client system by a graphical user interface. 

16. An article comprising a machine readable medium 
having a plurality of machine readable instructions, wherein 
when the instructions are executed by a processor of a client 
system the instructions subscribe the client system to base 
layers of multiple sources of multimedia data, subscribe the 
client system to an enhancement layer of a selected one of 
the sources, control the reception of the base layers and the 
enhancement layer in a layered stream over a fixed band- 
width communications path at the client system, display the 
base layer and the enhancement layer of the selected source 
in a first portion of a display of the client system, display the 
base layer of at least one non-selected source in at least one 



output signals for the second source in a second portion of 45 other portion of the display, unsubscribe the enhancement 



the display, unsubscribe the second layer of the first source, 
and to subscribe the processor to a second layer of the 
second source. 

9. The article of claim 8, further comprising instructions 
to subscribe the processor to a third layer of the first source, 
the third layer comprising audio data signals, to receive data 
signals corresponding to the third layer in a second stream 
over the fixed bandwidth communications path, and to 
produce audio output signals corresponding to the received 
data signals for the third layer. 

10. The article of claim 8, further comprising instructions 
to display the output signals for the first source in the second 
portion of a display, and to simultaneously display the output 
signals for the second source in the first portion of the 
display. 

11. The article of claim 10, further comprising instruc- 
tions to unsubscribe the second layer of the second source; 
and to subscribe the processor to the second layer of the first 
source. 

12. A method of streaming digital multimedia data signals 



50 
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layer of the selected source and subscribe to an enhancement 
layer of a newly selected source. 

17. The article of claim 16, further comprising instruc- 
tions for subscribing to an audio layer of the selected source, 
receiving the audio layer at the client system, and producing 
audio output signals in synchronization with the displaying 
of the base and enhancement layers of the selected source. 

18. The article of claim 16, further comprising instruc- 
tions for 

receiving the base layers of the sources and the enhance- 
ment layer of the newly selected source; 

displaying the base layer and the enhancement layer of the 
newly selected source in the first portion of the display; 
and 

substantially simultaneously displaying the base layer of 
at least one non-selected source in at least one other 
portion of the display. 

19. The article of claim 16, further comprising instruc- 
tions for controlling subscription of the enhancement layer 



of multiple sources from at least one server system in a 65 by a user of the client system by a graphical user interface, 
layered stream over a fixed bandwidth communications path 

in a network to a client system comprising: * * 
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